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Drawings 



1 . The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(5) because they do not include the 
following reference sign(s) mentioned in the description: reference numbers 131 and 133 (see page 11, line 12 of the 
applicant's disclosure). A proposed drawing correction or corrected drawings are required in reply to the Office 
action to avoid abandonment of the application. The objection to the drawings will not be held in abeyance. 



Specification 

2. The disclosure is objected to because of the following informalities: 

i. On page 11, line 21, the applicant refers to band 134. It is clear from the applicant's 
disclosure that reference to band 130 was intended. 

ii. The following conditional statement is taken from page 14, lines 27-28 of the applicant's 
disclosure: 

If (k,l) is not alive: 

Set Dkj = [Diminjmin + o)] in the distance image. 
The applicant does not define alive as it used here, nor can it be inferred from the 
disclosure what is meant by alive. 
Appropriate correction is required. 



Claims 

Rejections under U.S.C. § 103(a) 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if 
the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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4. Claims 1-7, 12, 13-15, 21, and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable over Dickie 
(U.S. Patent 5999651), in view Geiger, et al. ("Dynamic Programming for Detecting, Tracking, and Matching 
Deformable Contours"), in further view of Sethian ("Level Set Methods and Fast Marching Methods"). 

5. The following is with regard to claim 1 . Dickie discloses a method of tracing an object in a video image 
frame that is one of a sequence of video image frames. The method involves interactively defining an object 
boundary with respect to the current frame. This method is comprised of the following steps: 

i. A user inputs a set of control points (i.e. seed points and free points). See Dickie column 
5, lines 15-35. 

ii. An edge of minimal cost that spans two adjacent control points is derived (and includes 
those control points). The derivation of such edges is repeated for each set of adjacent 
control points so that the edges together form a minimal-cost closed contour that 
represents the boundary of the object of interest with respect to the current frame. As a 
result of how the contour is constructed, that closed contour includes each of the control 
points. See Dickie Figs. 3 and 5, column 2, lines 56-59 (Summary of Invention), column 
5, lines 28-51. 

Generally, allowing the user to input or select control points, in the manner just described, entails that provisions be 
made to further allow the user to modify or update that set of points (e.g. by adding or moving control points). The 
method of Dickie provides the user with a means to modify or update the set of input control points. A minimal-cost 
closed contour conjoining each control point of the set of modified control points is reevaluated. See Dickie, column 
8, line 4-6, lines 34-54, and Figs. 8a-b. This clearly addresses the aspects of applicant's claim 1 relating to the input 
(receipt) of control points, the updating of control points, and the derivation of a closed contour connecting those 
points. However, the control points that are input to Dickie's method are supplied by the user and are not necessarily 
the control points used to derive the boundary of the object of interest with respect to a prior frame. Furthermore, 
Dickie does not employ or suggest the usage of a restricted band about the object boundary that constitutes a domain 
of the image within which the minimal-cost closed contour is derived. 
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6. Geiger, et al., on the other hand, disclose a method for the detection and tracking of contours in consecutive 
frames wherein a minimal-cost boundary contour obtained in a previous frame is sampled and applied as the initial 
points for the next (i.e. current) frame (Geiger, et al. page 294, section LB: "Tracking" and paragraph 3 of Section 
III, "Tracking Deformable Contours" in the attached errata). See Geiger, et al., Section III, "Tracking Deformable 
Contours" of attached errata. Given the inherent similarity of the methods of Geiger, et al. and Dickie, as well as the 
similar context in which they are applied (i.e. in sequences of image frames), it would have been straightforward for 
one of ordinary skill in the art to modify the method of Dickie to receive as input those points sampled from the 
boundary contour of the prior frame, as opposed to points inputted manually by the user. It would be clear to one of 
ordinary skill in the art that by performing such a modification, one obtains a method capable of automatically 
detecting contours representing the boundary of an object with respect to subsequent image frames, given a contour 
associated with that object in the first image frame (Geiger, et al., paragraph 1 of Section III, "Tracking Deformable 
Contours" found in the attached errata). Clearly, this would, in turn, limit the amount of user intervention required to 
trace or extract objects from the image frames comprising a sequence of image frames. Thus, given this clear 
advantage and the straightforwardness of modification, it would have been obvious to one of ordinary skill in the art, 
at the time of the applicant's claimed invention, to modify the object tracing method of Dickie to accept as input the 
points sampled from the boundary contour of the prior frame, as suggested by Geiger, et al. 

7. The combined teachings of Dickie and Geiger, et al. do not suggest the usage of a restricted band about the 
object boundary that constitutes a domain of the image within which the minimal-cost closed contour is derived. 
Sethian, however, discloses a so-called "Narrow Band Level Set Method" in which a narrow band is placed around 
an initial front (e.g. a closed contour). This narrow band represents a restricted computational domain about the front 
(contour) and the front (contour) will propagate only within the bounds of the band. See Sethian, pages 80-85 and 
Fig. 7.5 for details. In this regard, the narrow band of Sethian is analogous to the restricted area defined in claim 1. It 
should be noted here that the applicability of the Narrow Band Level Set Method to image segmentation - 
particularly with respect to active contour methods - is well known in art. Sethian discusses this in detail in Section 
17.2 (pages 218-225). According to Sethian, by limiting the computational domain in this manner, the overall 
computational cost may be reduced substantially, resulting in an increase in speed. See Sethian pages 80-81 and last 
paragraph of page 83. Given the clear advantage, in terms of computational speedup, obtained by limiting the 
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computational domain to a narrow band about the aforementioned contour, as well as the demonstrated applicability 
of the Narrow Band Level Set Method to analogous image segmentation methods, it would have been obvious to 
one of ordinary skill in the art, at the time of the applicant's claimed invention, to incorporate the teachings of 
Sethian, with regard to limiting the computational domain to a narrow band about the front or contour of interest, 
into the combined teachings of Dickie and Geiger, et al., discussed above. The teachings of Dickie, Geiger, et al. and 
Sethian, combined in the manner just described, address all aspects of the applicant's claim 1. 

8. The following is with regard to claim 2. As previously mentioned, Dickie's method allows the user to 
modify a set of control points by either adding a control points or moving a preexisting control point. See Dickie, 
column 8, line 4-6, lines 34-54, and Figs. 8a-b. 

9. The following is with regard to claim 3. With regard to the Narrow Band Level Set Method, Sethian 
describes a process called "re-initialization", wherein the narrow band surrounding the front (contour) is rebuilt 
when the said front (contour) approaches either of the band's boundaries (Sethian page 83). Re-initialization can 
also be applied when new points are added (Sethian page 85, last paragraph). It should be clear that, in the case that 
a point should move outside of the bounds of the narrow band, re-initialization would also be applied. Thus, the re- 
initialization process would allow a method based on the teachings of Dickie, Geiger, et al, and Sethian, combined 
in the manner discussed above, to respond to user input in a manner similar to that which is claimed in claim 3. The 
arguments presented above with respect to claims 1 and 2 are applicable to claim 3. See paragraphs 5-8 above. 

10. The following is with regard to claim 4. In the image segmentation method of Dickie, each pixel of the 
image, or frame, is analyzed to determine certain characteristics of that pixel with regard to the neighboring pixel. 
These characteristics include an image gradient magnitude - i.e. for image I(x,y), the image gradient magnitude is G 
= ||VI(x,y) || (Dickie, column 4, lines 49-60). The examiner takes Official Notice that the edge energy is typically 
defined to be proportional to the square of the image gradient magnitude (e.g. edge energy E edge = ||VI(x,y) || 2 ). 
Clearly, the image gradient and edge energy are mathematically similar values. Moreover, according to equation 2 
of Dickie, for certain values of the gradient response gamma, X h the edge energy is evaluated. Specifically, for 
X\-2 and pixels p and q, 



Halfcostfp,^ = D(p,q)(A 2 + A 3 + A 4 -(1 - A 0 )G(p) 2 • Multiplier^, q)) 
Halfcostfo/7; = D(q,p)(A 2 + A 3 + A 4 - (1 - A 0 )G(q) 2 • Multiplier^, /?)) 
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where G(p) 2 =||VI(p) || 2 =E edge (p) and G(q) 2 =||VI(q) || 2 =E edge (q). The values, Halfcost(p,q) and Halfcost(q,p), and, 
hence, the respective edge energies, are later used to derive an edge of minimal cost between two control points. 
Again, the derivation of such edges is repeated for each set of adjacent control points so that the edges together form 
a minimal-cost closed contour that represents the boundary of the object of interest with respect to the current frame. 
See Dickie, column 6, lines 23-48 and column 7, lines 40-51. Thus, Dickie addresses the all the limitations of the 
applicant's claim 4. 

11. The following is with regard to claim 5. The invention of Dickie displays the current digital frame and the 
minimal-cost closed contour that represents the boundary of the object of interest with respect to the current frame. 
See Dickie Figs. 8a-b and column 8,lines 34-45. Allowing the user to improve the trace of the object with respect to 
the current frame entails displaying to the user the trace (e.g. Dickie Fig. 8a, reference number 83) relative to the 
object of interest (e.g. Dickie Fig. 8a, reference number 81) as it appears in the current frame (e.g. Dickie Fig. 8a, 
reference number 80). 

12. As mentioned above, with regard to claim 2, the invention of Dickie supports both the addition of new 
control points into the set of control points and the moving of preexisting control points from the set of control 
points. Furthermore, note that Dickie suggests the usage of a pointing device such as a mouse as means to interact 
with the control points. See Dickie column 5, lines 23-26 and 29-33. Thus, Dickie addresses all aspects of claim 5. 

13. The following is with regard to claim 6. Again, the re-initialization process discussed by Sethian can be 
used to redefine the narrow band (restricted area) so as to encompass points which have moved or been added 
outside the boundary of the narrow band (restricted area) due to user intervention. The reader is referred to the 
discussion in paragraph 8 of this document regarding Serbian's re-initialization process, as it is applicable to claim 6. 

14. The following is with regard to claim 7. As discussed earlier, the image segmentation method of Dickie 
allows the user to both move preexisting control points from the set of input control points and to add control points 
to that set. Furthermore, as stated in column 8, lines 4-6, 29-54 of Dickie and shown in Figs. 8a-b of Dickie, when a 
control point is moved or added, the minimal-cost edges between it and the two adjacent control points from the set 
of input control points are recomputed. This clearly addresses what is being claimed in applicant's claim 7. 

15. The following is with regard to claim 12. The following is an excerpt taken from Geiger, et al. (see Geiger, 
et al., the first to third paragraphs under Section III, "Tracking Deformable Contours" in the attached Errata): 



# • 
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Suppose that we are given a sequence of images ... and a deformable contour associated with an object in the first frame. 
How can we use this information to automatically detect the subsequent contours? 

. ..[T]he new contour should be close to the previous one... First, we simply copy the contour to the next frame and sample 
at the extrema points ... We consider these points to be the input, much like the ones provided by the user in the initial 
frame. 

It should be clear from this excerpt that the contour of a previous image frame (e.g. the first image frame) serves as 
the object boundary estimate from a prior image frame, in accordance with the applicant's claim 12. Therefore, in 
tracking the contour representing the boundary of the object of interest, an object boundary estimate from a prior 
frame is received. Further note that, according to the excerpt above, a set of control points from the prior image 
frame (i.e. the points sampled from the previous contour) are also received. Geiger, et al. also show, in the above 
excerpt, that the object boundary estimate from a prior frame, itself, provides an initial estimate for the contour that 
represents the boundary of the object of interest with respect to the current frame. This, of course, facilitates the 
derivation of the minimal-cost closed contour that represents the boundary of the object of interest with respect to 
the current frame. Again, one of ordinary skill in the art could construct a narrow band about the received object 
boundary estimate, in the manner of Sethian's Narrow Band Level Set Method (see paragraph 7 above), thus, 
defining a restricted area in accordance with claim 12.. The motivation to do so is the same as that which was 
presented above for claim 1. Thus, given the teachings of Geiger, et al., as they are present here and the teachings of 
Sethian, as applied to claim 1 , it would have been it would have been obvious to one of ordinary skill in the art, at 
the time of the applicant's claimed invention, to construct an image segmentation method, in accordance with claim 
1, that receives an object boundary estimate from a prior frame and defines a narrow band restricted area about that 
boundary estimate. Arguments presented above with regard to claim 1 further support this conclusion. See 
paragraphs 5-7 above. 

16. Note that all aspects of the apparatus put forth in claim 13 are addressed by the method of claim 1. 
Therefore, with respect to claim 13, arguments analogous to those presented for claim 1, are applicable. See 
paragraphs 5-7. 

17. Note that all aspects of the apparatus put forth in claim 14 are addressed by the method of claim 3. 
Therefore, with respect to claim 14, arguments analogous to those presented for claim 3, are applicable. See 
paragraph 9 above. 
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18. Note that all aspects of the apparatus put forth in claim 1 5 are addressed by the method of claim 6. 
Therefore, with respect to claim 15, arguments analogous to those presented for claim 6, are applicable. See 
paragraph 13. 

19. Note that all aspects of the apparatus put forth in claim 16 are addressed by the method of claim 7. 
Therefore, with respect to claim 16, arguments analogous to those presented for claim 7, are applicable. See 
paragraph 14 above. 

20. Note that all aspects of the apparatus put forth in claim 21 are addressed by the method of claim 12. 
Therefore, with respect to claim 21, arguments analogous to those presented for claim 12, are applicable. See 
paragraph 15 above. 

21. It would be obvious to one of ordinary skill in the art that all the processing done as per claim 1 3 could be 
done using a single processor. Therefore, with regard to claim 22, arguments analogous to those presented for claim 
13 are applicable. See paragraph 16 above. 

22. Claims 8-10, and 17-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Dickie, Geiger, et 
al., and Sethian as applied to claim 1, and in further view of Sun, et al. (U.S. Patent 65461 17). 

23. The following is with regard to claim 8. As shown above, the teachings of Dickie, Geiger, et al. and 
Sethian, when combined in the manner described above, address all aspects of the applicant's claim 1. However, the 
teachings of Dickie, Geiger, et al. and Sethian do not address the following aspects of the applicant's claim 8: 

a. determining a distance from one control point to an adjacent control point; 

b. applying a first set of rules for deriving a path or edge from the said one control point to 
the said adjacent control point, if the distance between these points is less than a 
threshold distance; 

c. applying a second distinct set of rules for deriving a path or edge from the said one 
control point to the said adjacent control point, if the distance between these points is 
greater than a threshold distance. 

24. Sun, et al. disclose a video object segmentation method similar to the image segmentation methods of 
Dickie and Geiger, et al. In deriving a minimal-cost closed contour that represents the boundary of the object of 
interest with respect to the current frame, the distance between input edge points (control points) is compared to a 
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threshold distance and a respective set of actions is performed depending on whether the distance is greater than a 
threshold distance or less than a threshold distance. Note that, when viewed within the context of the applicant's 
disclosure and the prior art, the set of actions referred to above can reasonably be interpreted as being a set of rules. 
See Sun, et al. column 17, lines 45-67 to column 18, lines 1-12 and Fig. 11, reference numbers 194-198. 
25. Calculation of the distance between two adjacent edge (control) points is trivial. Given this and the inherent 
similarity between the methods of Sun, et al. and the methods of Dickie and Geiger, et al., the integration of the 
threshold distance comparison of Sun, et al. into the method, derived from the combined teachings of Dickie, 
Geiger, et al., and Sethian (see paragraphs 5-7 above), can be executed with ease by one of ordinary skill in the art. 
It would also be clear to one of ordinary skill in the art that by eliminating control points (i.e. performing the first set 
of rules) when the distance between adjacent control points is less than the first threshold (as suggested by Sun, et 
al.), one reduces the overall computational load of deriving a minimal-cost closed contour that represents the 
boundary of the object of interest with respect to the current frame, while effectively maintaining the accuracy of 
that representation. This is true because, when this threshold distance is chosen to be sufficiently small, as suggested 
by Sun, et al., it can be assumed that the nature of the object boundary does not change substantially over that 
threshold distance. Thus, regions between control points with distances less than the first threshold distance can be 
neglected. In addition, it would be clear to one of ordinary skill in the art that by adding additional control points via 
interpolation of two adjacent control points (i.e. performing the second set of rules) when the distance between the 
said control points is greater than the second threshold distance (as suggested by Sun, et al.), one improves the 
accuracy of the minimal-cost closed contour that represents the boundary of the object of interest with respect to the 
current frame. This is true because, in general, the nature of the object boundary can change substantially over large 
regions of the given image frame. Therefore, the insertion of additional control points between adjacent control 
points with large distances spanning them makes the image segmentation method more sensitive to the local features 
of the object boundary. Given the clear advantages of introducing the teachings of Sun, et al. into the teachings of 
Dickie, Geiger, et al, and Sethian, combined in the manner described above, as well as the ease with which these 
teachings can be combined, it would have been obvious to one of ordinary skill in the art, at the time of the 
applicant's claimed invention, to use the threshold distance comparisons described by Sun, et al. in the image 
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segmentation method obtained by combining the teachings of Dickie, Geiger, et al., and Sethian, in the manner 
described above. 

26. The following is with regard to claims 9 and 10. It should be clear from the discussion in the previous 
paragraph that the second set of rules (i.e. adding additional control points between adjacent control points by 
interpolating the said adjacent control points and then deriving the minimal-cost closed contour that represents the 
boundary of the object of interest with respect to the current frame based on the new set of control points) produces 
a more accurate representation of the object boundary between two adjacent control points than the first set of rules 
(i.e. eliminating control points and then deriving the minimal-cost closed contour that represents the boundary of the 
object of interest with respect to the current frame based on the new set of control points) when the distance between 
the adjacent control points of interest is large (e.g. greater than the threshold). Thus, the discussion in the previous 
paragraph is applicable to claim 10. 

27. Regarding claim 9, one of ordinary skill in the art would realize that applying the second set of rules - in 
particular, inserting interpolated control points between the two adjacent control points in question - may, when the 
distance between the adjacent points is sufficiently small, degrade the accuracy of a minimal-cost closed contour 
derived via the method taught by Dickie (or the combined teachings of Dickie, Geiger, et al. and Sethian). To see 
this, consider the following example. Let the threshold distance be equal to 2.5 pixels, as suggested by Sun, et al. 
(Sun, et al. column 18, line 5). Let the distance between the adjacent control points, i and j, be 2 pixels. Let X be the 
interpolated control point. The following figure is intended to be a section of the current image frame that illustrates 
this case: 

+ + + + + + + nisa neighbor of control point i 

+nnn+++ Xis the interpolated control point 

+niX+j+ + is an image pixel 

+ n n n + + + 

+++++++ Note that X is also a neighbor of i 

According to the image segmentation method of Dickie, a minimal-cost edge is constructed between adjacent 
control points. However, according to column 7, lines 45-51 and Fig. 5 of Dickie, no such edge is constructed 
between the control point i and the interpolated control point X, since clearly there are no pixels between these 
points. The contour derived to represent the object boundary would then include the edge formed by control points i 
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and X. The minimal-cost path between control points X and j is later derived. The edge spanning control points i and 
j would obviously include the interpolated control point X. This edge, however, may not be the optimal edge, since 
there may be some neighbor pixel, n, with a lower cost than the pixel X (i.e. the interpolated control point). In other 
words, the minimal-cost path between i and j may not include the pixel X. Therefore, the insertion of the 
interpolated pixel, X, can result in a contour, representing the object boundary with respect to the current image 
frame, that is not optimal. It should be evident from this discussion and the figure above that this degradation in 
accuracy or optimality is exacerbated when the distance between adjacent control points is small. 

28. The preceding paragraph demonstrates that, for a sufficiently small threshold distance, the second set of 
rules degrades the accuracy of the derived contour. As mentioned above, however, application of the first set of rules 
has a negligible effect on the accuracy of the derived contour. Therefore, in accordance with claim 9, the first set of 
rules is more accurate than the second set of rules in deriving a minimal-cost edge between adjacent control points, 
when the distance between those points is less than a certain threshold. 

29. Note that all aspects of the apparatus put forth in claim 17 are addressed by the method of claim 8. 
Therefore, with respect to claim 17, arguments analogous to those presented for claim 8, are applicable. See 
paragraphs 23-25 above. 

30. Note that all aspects of the apparatus put forth in claim 1 8 are addressed by the method of claim 9. 
Therefore, with respect to claim 18, arguments analogous to those presented for claim 9, are applicable. See 
paragraphs 27-28 above.. 

3 1 . Note that all aspects of the apparatus put forth in claim 19 are addressed by the method of claim 10. 
Therefore, with respect to claim 19, arguments analogous to those presented for claim 10, are applicable. See 
paragraph 26 above. 

32. Claims 1 1 and 20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Dickie, Geiger, et al, 
and Sethian as applied to claims 1, 5, and 13, and in further view of Guo, et al. ("New Video Object Segmentation 
Technique with Color/motion Information and Boundary Postprocessing"). 

33. The following is with regard to claim 1 1. As shown above, the teachings of Dickie, Geiger, et al. and 
Sethian, when combined in the manner described above, address all aspects of the applicant's claim 1 . However, 
Dickie, Geiger, et al. and Sethian do not teach morphologically dilating the contour connecting the control points in 
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order to obtain a restricted area about that contour. Guo, et al. introduce a video object segmentation technique, 
wherein a variable width object boundary is constructed by performing morphological dilation on received motion- 
tracked object boundaries. See Guo, et al. Section 3.2, "Motion Tracking and Spatial-Temporal Integration", Section 
4, "Boundary Postprocessing for Efficient Shape Coding", and Fig, 2. The object masks obtained from the spatial- 
temporal segmentation contain information based on the object boundary(s) relative to a prior image frame(s) and, 
thus, implicitly define a set of points in the current image frame corresponding to the object boundary(s) of a prior 
image frame(s). See the last paragraph on page 13 of Guo, et al. The variable-width boundary represents an area 
about the proposed boundary of the object (based on the received object mask[s]) wherein the final object boundary 
- i.e. the boundary of the object relative to the current frame - is derived. In this regard, the variable-width boundary 
of Guo, et al. is similar to the restricted area of the applicant's claim 1 . 

34. Given that morphological dilation can be easily and efficiently implemented (see, for example, the 
definition of the dilation operation on page 12 of Guo, et al.) and is typically applied to image data, it would be 
straightforward to one of ordinary skill in the art to apply morphological dilation, in the manner of Guo, et al., to 
received object boundary points or pixels. Furthermore, given the simplicity of the operation and its well-known 
dilative properties, it would be clear to one of ordinary skill in the art that morphological dilation provides a simple 
and efficient means to construct a band-like region (i.e. a dilation of the contour) around a contour representing an 
object boundary or any other contour, for that matter. Thus, given the simplicity of the morphological dilation 
operation and the ease with which it can be integrated into an image or video segmentation method, it would have 
been obvious to one of ordinary skill in the art, at the time of the applicant's claimed invention, to use morphological 
dilation, in the manner suggested by Guo, et al., to construct a band about a contour representing an object relative 
to a prior image frame, wherein that band represents a restricted area in which a contour, representing the object 
relative to the current frame, is derived. 

35. Note that all aspects of the apparatus put forth in claim 20 are addressed by the method of claim 1 1 . 
Therefore, with respect to claim 20, arguments analogous to those presented for claim 1 1, are applicable. See 
paragraphs 33-34 above. 
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